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ABSTRACT 


A study was conducted to determine whether the Maintenance 
Training Improvement Program (MTIP) has any relationship to 
unit performance of the west coast F-14 and E-2 squadrons. 
Using correlation analysis, the MTIP completion rate was 
compared with operational measures such as the Full Mission 
Capable (FMC) rate, the number of no-defect (malfunction code 
A-799) maintenance actions, and the Direct Maintenance 
Manhours per Flight Hour (DMMH/FH). A moderate positive 
correlation was found between the MTIP completion rate and the 
FMC rate, and a moderately weak negative correlation was found 
between the MTIP completion rate and the number of A-799 
Maintenance actions. There was no correlation found between 
the MTIP completion rate and the DMMH/FH. These relationships 
have provided some insight with encouraging implications for 


further research to assess the existing training program. 
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I. INTRODUCTION 


The world of naval aviation has been in existence for over 
eighty years since Eugene Ely, a civilian pilot, flew a 
Curtiss biplane from a wooden ramp built on the forecastle of 
USS Birmingham (Johnson, 1981). Since then, naval aviation 
has progressed from canvas framed aircraft with bicycle tires 
and wooden propellers to highly sophisticated, complex 
airborne weapons systems. The complexity of modern aircraft 
in the U.S. Navy requires high levels of knowledge and skill 
of our technicians who must maintain the various high-tech 
systems so that the aircraft can perform their intended roles 
such as air-to-air combat, ordnance delivery, and early 
airborne warning, just to mention a few. 

It is reasonable to helieve that well-trained personnel 
perform better on their jobs than do less-well-trained 
personnel (Gibson and Orlansky, 1986, pp.5). _If this is so, 
the effects of better training should be observable in such 
indicators as the amount of direct maintenance manhours 
required per each flight hour (DMMH/FH), the number of 
components removed by the Organizational Maintenance Level 
("O" Level) technicians as defective that are found to operate 
properly when tested later (this error is reported as a 


Malfunction Code A-799), and the operational readiness 





posture--Full Mission Capable (FMC) rate--for particular 
aviation units. If the effects of training are not reflected 
in unit performance and operational readiness, then the 
existing training program needs to be either improved or 
replaced by an alternative program. Deficiencies in technical 
training and personnel qualifications could prevent a 
technician from acquiring the proper knowledge needed to 
perform at his or her top efficiency. The ourpose of aviation 
Maintenance training is to prepare sailors to perform the 
technical tasks necessary to assure the availability and 
proper functioning of aircraft weapons systers and support 


equipment. 


A. SCOPE OF STUDY 

In any naval aircraft community, shortages of trained 
personnel can pose a serious threat to naval air warfare 
operations. The Maintenance Training Improvement Program 
(MTIP) establishes an initial level of proficiency and 
provides a baseline for a tailored training program for 
improving the knowledge of technicians. However, the program 
has not been tested to determine whether it is ultimately 
helping an aviation squadron to maintain or improve its 
performance at a competitive level. This thesis examines the 
relationship between i::7IP and performance indicators such as 
DMMH/FH, Code A-799, and FMC rates. The main thrust of this 


study is to determine whether there is any correlation between 





MTIP traiuing and the operational readiness of the Tomcat (F- 
14) and Hawkeye (E-2) aircraft communities stationed at Naval 
Air Station, Miramar, California. 

This research is based on the data collected from the 
Flight Activity, Inventory and Readiness Report; the 
Individual Maintenance Action Record; and the Monthly MTIP 
Completion Rates from the Fighter Airborne Early Warning Wing, 


U.S. Pacific Fleet (FITAEWWINGPAC) . 


B. THESIS ORGANIZATION 

Chapter II discusses the evaluation methods available to 
authors, and reviews and summarizes the previous literature 
reporting the performance measures used to evaluate the 
effectiveness of maintenance training. Examples are given of 
how various methods of measurement were performed. 

Chapter III provides an in-depth background of the Naval 
Aviation Maintenance Training Program (MTIP). Policies, 
procedures, requirements, minimum standards, and assigned 
responsibilities of the MTIP training are discussed. 

Chapter IV presents the data collected from the Equipment 
Condition Analysis Report and the MTIP Completion Rates. Only 
relevant information from this raw data should be extracted, 
compiled, and displayed in a logical format. Also included in 
this chapter are the analyses that describe the strength of 


the statistical relationships between the selected performance 


indicators and MTIP completion rates. 





Finally, Chapter Vv contains conclusions and 


recommendations about the impact and effectiveness of MTIP 


training on the operational readiness of aviation squadrons. 





II. BACKGROUND 


A. INTRODUCTION 

The purpose of aviation maintenance training is to prepare 
sailors to perform the technical tasks necessary to assure the 
availability and proper functioning of aircraft weapons 
systems and support equipment. Aviation maintenance training 
programs should be considered dynamic entities that slowly 
accomplish their purpose in meeting predesigned objectives. 
Without systematic evaluation, there is no feedback to provide 
the information necessary to improve the programs through the 
selection, adoption, and modification of new or existing 
instructional designs. It is important to use the most 
systematic procedures available that fit the particular 
setting being investigated, to control as many variables as 
possible, and to recognize the limitations of the design being 


used. This chapter examines the variables that can affect an 


evaluation and describes the various evaluation methodology 


available to researchers, following a review of previous 


studies. 


B. FACTORS THAT AFFECT EVALUATION 
The need for a systematic evaluation of the Maintenance 
Training Improvement Program (MTIP) implies a need for 


measurement techniques that clearly establish the relationship 





between training and performance on-the-job. In training 
settings that can be tightly controlled, evaluation can be 
straightforward. For example, on-site training in a 
manufacturing setting is conducive to evaluation. 
Conceivably, personnel could be tested on work skills before 
training (pretesting), training would be completed, and 
workers would be tested again after training (posttesting). 
In such a setting it is likely that job performance measures 
would be readily available and, therefore, posttest 
performance could be compared to pretest performance to 
evaluate the effectiveness of the training. Unfortunately, 
most real-world training occurs in settings where such tight 
control--and perfect evaluation--is not possible. Two areas 
that make training evaluation problematic are discussed here: 
(1) threats to internal validity, and (2) threats to external 
validity. Threats to validity can be controlled through 
selection of the appropriate experimental design, which is 
also discussed here. 
1. Internal Validity 
Internal validity refers to whether the training 
program makes a difference within the instructional setting. 
That is, if a test is given immediately after training, e.g., 
in the classroom, and scores are higher than they were on a 
test given immediately before training, can those differences 


be attributed to the training--did learning occur? Of course, 


the real concern is with transfer of learning to on-the-job 
performance, that is, did learning occur and also produce 
better performance? However, the issue of transfer of 
training cannot be dealt with unless internal validity has 
been established. 

Variables other than instructional events that affect 
the results of training are threats to internal validity. 
Examples of son of these variables have been described by 
Goldstein (1993): 

a. History 

This variable refers to specific events occurring 
between two measurements that could influence or provide 
alternative explanations for the results of a training 
evaluation. For example, with the outbreak of the Gulf War, 
a few squadrons were required to take the posttest without 
completing the entire training program in order to meet the 
new deployment schedule. This unplanned change may produce 
significant differences that have no relationship to the 
Material presented in the original instructional program. 

b. Maturation 

This refers to both biological or psychological 
effects that vary with the passage of time (getting older or 
loss of interest in the program between two measurements) and 
affect performance for reasons unrelated to a training 


program. 


c. Testing 
A pretest may influence the results of the 
posttest. Simply becoming familiar with the types of 
questions on the pretest may enable a maintenance technician 
to concentrate on materials that may provide answers to the 
next test. 
da. Instrumentation 
Changes in grading standards, configuration changes 
with a new model/type of equipment, or even changing the 
evaluator can result in differences between two measurements 
that make it difficult to determine if learning occurred. 
e. Selection of Participants 
It can be difficult to determine the effects of 
training when participants bring differing amounts of 
knowledge to the learning situation that cannot be accounted 
for in the experimental design. For example, when only group 
performance can be used as the measure of training even though 
pretesting and posttesting for individuals are conducted 
(which is the case in the present research), differences could 
occur between the experience levels of different groups. 
However, this variable is best controlled by random selection 
of the entire population. 
f£. Interactions 
Any of the variables discussed may interact to 


create a threat to internal validity. The likelihood of this 


occurring is increased when training is evaluated outside of 
a controlled, laboratory setting. 

g. Rivalry between Participants 

When an evaluation is made public or when two units 
are assigned to a particular condition, the special effort 
based on motivation may wipe out a true reflection of how each 
group would ordinarily perform. 
2. External Validity 

External validity deals with the ultimate concern of 
training: not only did learning occur, but can it be 
generalized to other groups and settings? Most relevant here 
is the issue of whether training is related to on-the-job 
performance. Threats to external validity, again discussed in 
Goldstein (1993) include the following: 

a. Reactive Effect of Pretesting 

As noted earlier, a pretest can sensitize 

individuals to information presented in training with the 
results of improved posttest scores. In research studies, 
where evaluation often ta’<s place, pretesting may be used 
during the research--thereby enhancing the effects of 
experimental training--but not in the actual implementation of 


the training. 








b. Interaction of Selection and Experimental 
Treatment 
The characteristics of the group often determine 
the applicability of the findings for other purposes. The 
characteristics of maintenance technicians from one squadron 
May result in the training being more or less effective for 
them as compared to an another squadron. 
c. keactive Effects of Experimental Settings 
Controlled environments with special setups and 
equipment, which may be used in evaluation research, can lead 
to changes in behavior that will not be present when training 
is implemented in a non-research environment. 
3. Experimental Design 
A variety of experimental designs are available to 
evaluate the effects of training. The ideal design is one 
that would produce "clean" results, having eliminated all 
threats to validity. With such an ideal model one could 
conclude with complete confidence that observed results were 
a function of the training. This ideal model would require 
the availability a group who receives training (experimental 
group), matched to a group for comparison that did not receive 
training (control group), and the opportunity for pretesting 
and posttesting with objective performance measures. In real- 
world settings, the use of the ideal experimental design is 


rarely possible. Therefore, the experimental design is 











tailored to accommodate the constraints of the training and/or 
on-the-job setting. These modified designs will control 
threats to validity of the findings differently. While not 
always perfect, many researchers have discussed the importance 
of doing partial evaluation instead of avoiding it because of 
the inherent difficulties (Peterson, 1978). Some of the 
traditionally used designs are discussed here. 
a. The One-group Posttest Only Design 

Trainees are exposed to the training program 
without a pretest and then are tested upon completion of 
training. Without the pretest baseline and a control group to 
compare it with, it is not possible to ascertain any change as 
a result of the training program. This design is quite 
limited, because findings cannot be generalized to other 
areas. 

b. The One-group Pretest and Posttest Design 

The participants are given a pretest before the 
training program, and a posttest upon completion of training. 
This design provides a measure of comparison between 
performance before and after the training program. Again, 
without a control group, it is difficult to establish whether 
the change is due to the training program. Internal validity 


variables such as history, maturation, testing effects, and 


changes in instrumentation are not well controlled. 





c. Pretest and Posttest Control-group Design 
The participants are chosen at random and assigned 
randomly to the experimental group or control group. Each 
group is given a pretest and posttest, but only the 
experimental group is provided with the training program. 
Many of the internal validity threats are controlled. 
Variables such as history, maturation, and pretesting affect 
both groups equally. Differential treatment of participants 
in both groups must still be controlled by the evaluator. 
Pretesting does sensitize and affect participants. To control 
this sensitization, a third group may be introduced without 
taking the pretest, which is the intent of the next design. 
d. Solomon Four-group Design 
This procedure is designed to consider external 
validity variables. This method adds two groups that are not 
pretested--one with training, one without training. If the 
participants are randomly assigned to the four groups, this 


design makes it possible to compare the effects of pretesting. 


A comparison of the posttest for group 4, which was not 


exposed to pretesting or training, to the pretest scores of 

groups 1 and 2 makes it possible to analyze the combined 
effects of maturation and history. 
e. The Time-series Design 

This method is similar to the one-group posttest 


design, except that a series of measurements are taken before 











and after training. If there are no appreciable changes 
through successive measurements, it is unlikely that any 
effects will occur due to maturation or testing. The major 
internal validity threat with this method is the history 
factor where events may change as environmental changes and 
historical occurrences are not controlled by this procedure. 
This design does not control most of the external validity 
threats. Therefore, it is necessary to be sensitive to 
factors that might make results difficult to generalize to 
other groups. 
f£. The Nonequivalent Control-group Design 

This design is similar to the pretest and posttest 
control group design, except the participants are not assigned 
to the groups at random. This design utilizes groups already 
established such as class cohorts, where manipulating 
assignment to groups is not possible. If there is no 
alternative, this method is well worth using and is preferable 
to designs that do not have control groups such as the pretest 
and posttest group design. The more similar the two groups 
and their scores on the pretest, the more effective the 
control becomes in accounting for factors like history, 
pretesting, maturation, and instrumentation. This design is 
vulnerable to interactions between maturation, history, and 


testing because the participants were not randomly selected. 








And there is always a chance that they differ on some critical 
variable. 
4. Other Evaluation Models 

While variants on experimental design models are 
traditionally used for evaluating training, there are other 
approaches that can be used. 

a. Individual-difference Model 

This method uses statistical methods to measure the 

relationship between training and actual job performance. The 
emphasis here is on demonstrating that performance in training 
will predict performance on the job. Typically, correlation 
coefficients are used to measure the relationships of 
interest. The problem, of course, is that correlations 
measure relation, not causation. A strong correlation in this 
situation means that people who do well in training do well on 
the job. The correlation does not say that the training 
program caused people to do well on the job. This concept is 
relevant to this thesis and will be discussed further in the 
discussion of the results. - 

b. Content-validity Model 

This method relies on the assumption. that 

judgements can be made about the adequacy of the design of the 
training program. If the needs assessment has been done 
correctly and has been used to determine the knowledge, 


skills, and abilities (KSAs) taught in the program, then the 


14 


program is content valid. Effectively, the concept is that a 
good needs analysis automatically guarantees that the training 
is related to the KSAs needed from the job. The problem is 
that the method does not address whether the KSAs have been 
learned or, in fact, transferred to the job. Assuring content 
validity and performing correlational analyses would provide 


a more adequate means of evaluating training. 


C. REVIEW OF PREVIOUS STUDIES 

There has never been a study conducted to evaluate the 
Maintenance Training Improvement Program (MTIP). However, a 
few studies have examined operational readiness measures such 
as malfunction code A-799 no-defect rate, sortie completion 
rate, and elapsed maintenance time. These studies demonstrate 
some of the approaches that have been taken to evaluate 
training in real-world settings. Examples of these are 
summarized below: 

1. Orlansky and String (1981), reviewed a group of seven 
studies across all military services. The evaluation method 
used in this review does not fit into any models described in 
this chapter. Basically, information was collected and found 
that non-faulty parts were removed in 4 to 43 percent of all 
corrective maintenance actions accounting for 9 to 32 percent 
of all maintenance man-hours. Non-faulty parts are those that 
were removed but found not to be defective when received for 


repair by the next level of maintenance activity. The 
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researchers concluded that these data offer strong evidence 


that maintenance technicians may conduct maintenance in an 
inappropriate and inefficient manner. The researchers offer 
no inference to identify the factors that may lead to 
inappropriate maintenance. However, training is considered a 
possible factor contributing to the high A-799 rate. 

2. Horowitz and Angier (1985) found that the experience 
level, as indicated by the paygrades of the maintenance 
personnel, is the most consistent predictor of readiness. 
Using the number of Navy A-7 aircraft flights off a carrier in 
a quarter as a measure, a positive relation between payyrades 
and the number of sorties completed per quarter was observed. 
The researchers concluded that formal training and on-the-job 
experience/learning have an observable and meaningful impact 
on operational measures. However, the evaluation design used 
in this study again does not resemble any of the models 
described in this chapter. No pretest or posttest nor control 
group was employed in this study. Again, a relationship was 
Measured--not causation--and the results could have been 
caused by variables such as history, maturation, or 
motivation. Without controlling these factors, ascertaining 
any change in the performance as a result of training is not 
possible. 

3. Johnson, McConnell, and Mirdock (1983), used data from 
the Air Force Consolidated Data System for F-16 aircraft and 


found that speed in accomplishing maintenance tasks was 
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related to the completion of formal school training provided 


by the Field Training Detachment (FTD). No pretest or 
posttest was conducted to measure the effects before and after 
the training program. Variables such as history and 
maturation were not positively controlled. However, control 
groups were used to evaluate the effects of formal training. 
Maintenance work centers with over 60 percent FTD-trained 
personnel were compared with work centers with less than 60 
percent FTD-trained personnel. The data collected were 
limited to Work Unit Codes that are used cto identify the 
system, subsystem, assembly and component, etc., on which the 
Maintenance task is performed. Formal training had a greater 
effect on reducing the time needed to perform maintenance than 
did the number of maintenance actions performed by a worker. 
The study finds that work hours used to complete a task is a 
meaningful and useful productivity measure that is sensitive 
to differences in training backgrounds and methods. 

As evidenced in real-world settings of these studies, 
there are no proven off-the-shelf methodologies for evaluating 
maintenance training programs where both internal and external 
threats can be tightly controlled. The primary goals of the 
Maintenance Training Improvement Program (MTIP) are to raise 
technical knowledge levels, to contribute to unit performance 
and to enhance and imp.ove existing formal training. Without 
systematic evaluation, there is no feedback to provide 


information necessary to improve this program. 
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The next chapter examines the Maintenance Training 
Improvement Program in detail. Policies, procedures, 
requirements, minimum standards, and assigned responsibilities 
of the MTIP training are discussed. The methodology for this 
thesis study relative to the concepts discussed here are 


explained in Chapter IV. 
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III. MAINTENANCE TRAINING IMPROVEMENT PROGRAM 


A. INTRODUCTION 
The Assistant Chief of Naval Operations (Air Warfare), as 
program sponsor, provides the following overall program 


direction: 


The Maintenance Training Improvement Program (MTIP) is 
a training management system which shall be implemented 
throughout naval aviation (Chief of Naval Operations, 
OPNAV Instruction 4790.2E, June 1990). 


Since the mid-80s, MTIP has become an integral part of the 
overall aviation maintenance training program. It is 
compatible with and supports the formal technical training 
programs in naval aviation. MTIP is designed to supplement 
formal training provided by the Naval Air Maintenance Training 
Group (NAMTD) and Fleet Readiness Aviation Maintenance 
Personnel (FRAMP). All personnel should receive formal 
training prior to, or upon assignment to, a maintenance 
production work center. In cases where formal training is not 
received, MTIP testing is the only assessment tool available 
to determine an individual’s knowledge level. The primary 
goals are to raise technical knowledge levels, and to enhance 


and improve existing formal training. 
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The Maintenance Training Improvement Program (MTIP) is the 
primary method used to identify personnel training 
deficiencies at the organizational and intermediate 
maintenance levels. Through diagnostic testing procedures 
consisting of standardized question and answer banks from 
which computer generated tests are assembled, a qualitative 
assessment of individual technical knowledge can be made. The 
questions are designed to test knowledge of systems for each 
occupational specialty/rating required of a maintenance 
technician on the job. As deficiencies are identified, 
refresher or remedial training is conducted. Such assessments 
not only help to improve existing training materials and 
courses but also: 

1. Establish the baseline knowledge level of an aviation 
community such as F-14 TOMCAT or E-2 HAWKEYE, or a particular 
Squadron, work center, and/or an individual technician 
compared to his or her peers. 

2. Concentrate training efforts where they are most 
needed, according to the established baseline. 

3. Provide a training program tailored to individuals. 

4. Correct specific systems knowledge deficiencies prior 
to extended deployment while dedicated training resources are 
available. 

5. Provide a feedback loop across all phases of training, 
and interface with all echelons from the user through the 


functional wing (COMFITAEWWINGPACINST 1540.10). 
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MTIP training primarily relies on static media such as 


printed technical manuals, workbooks, and slides. There is 
also some use of films and video, but the use of advanced 
instructional technologies (e.g., computer-based training) has 
not been incorporated. MTIP provides units with some 
flexibility to direct their own inservice training program and 
yet be able to compare the outcomes to established standards. 
However, as the rate at which a squadron operates increases, 
the program is often seen as administratively burdensome and 
secondary to operational requirements (Ledeboer, 1988). The 
operational tempo and commitments often become the driving 
factors for accomplishing the program. 

MTIP is a training management system used to identify 
aviation personnel training deficiencies throughout the Navy, 
but the ultimate goal of MTIP is to improve the operational 
readiness of aviation squadrons. It is essential that 
training of aviation maintenance personnel be afforded the 
same level of attention as flight crew training. An effective 
MTIP requires high emphasis and dedicated effort at every 
level of command (COMFITAEWWINPACINST 1540.1J). This thesis 
examines whether there is any relationship between MTIP and 
the operational readiness of F-14 and E-2 squadrons within the 
Fighter Airborne Early Warning Wing, Pacific located at Naval 


Air Station, Miramar. 
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B. TEST REQUIREMENTS 

All sea and shore duty aviation personnel (E-6 and below), 
including staff and administrative support services such as 
Quality Assurance Representatives (QAR), Aviation 
Supply/Storekeepers (AK), and Aviation Administrationman (AZ) 
who are assigned to an aviation maintenance department work 
center, are administered MTIP testing. QARS consist of 
senior, highly experienced technicians from various work 
centers (e.g., Power Plants, Avionics, etc.). They are hand- 
picked by their divisions, qualified by the Quality Assurance 
Division, and officially designated by the Maintenance Officer 
in accordance with the requirements set forth in OPNAVINST 
4790.2E. They are detached from their work center and 
assigned to serve as staff members responsible to the Quality 
Assurance Division. All QARs are tested not only in the 
technical knowledge of their occupational specialty but also 
for skills in quality assurance administration. All E-5s and 
E-6s assigned to production work centers in supervisory 
positions are tested in their occupational specialty, as well 
as on Naval Aviation Maistenance Program (NAMP) supervisory 
knowledge. Tests on NAMP supervisory knowledge are not 


required of QARs, AKs, or AZs. 
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1. Sea Duty Personnel 


For aviation squadrons deploying as part of a Carrier 


Air Wing Group (CVW), MTIP testing is scheduled by the parent 


functional wing Maintenance Training Team. There are two 
required tests for each complete training cycle (from 
completion of a deployment to the start of a next deployment), 
which varies in length in accordance with the Commander Naval 
Air Force United States Pacific Fleet (CNAP) deployment 
schedule and averages one to one and half years. The two 
required tests are discussed in the following paragraphs: 
a. Post-deployment Test (CM) 

Testing materials are sent by the functional wing 
Maintenance Training Team to the Carrier Air Wing Group (CAG) 
Maintenance Officer for administration of post-deployment 
testing by the squadron Assistant Maintenance Officer. Those 
personnel who will not take part in the next deployment are 
not required to take the CM tests. However, any prudent 
Maintenance officer will undoubtedly strongly encourage all 
personnel to take the post-deployment test for the sake of 
providing continuous training until an individual is ready for 
transfer. CM tests may be completed up to forty-five days 
prior to completion of the deployment, but no later than 
thirty days after return from deployment (COMFITAEWWINGPACINST 
1540.13). Completing the post-deployment test early enables 


the packages to be graded, analyzed, where deficiencies are 





noted and compared, and presented to the Commanding Officer of 
each squadron at the earliest possible time. An aviation 
unit’s turnaround (from completion of a deployment to the 
start of a next deployment) Maintenance Training Improvement 
Program could commence immediately upon receiving the results. 
The various MTIP reports provided initially by the Maintenance 
Training Team (MTT) help the squadron maintenance training 
officer or chief petty officer to manage the training program 
by tailoring each individual’s needs in accordance with 
his(her) areas of deficiencies. Whenever an updated status of 
the training requirements is desired, MTIP reports may be 
requested via the Aviation Training Support System (ATSS) 
computer terminal located at each squadron. 
b. Pre-deployment Test (CP) 

CP tests must be completed no later than ninety 
days prior to a deployment. All personnel (E-6 and below) who 
are taking part in the deployment are administered a pre- 
deployment test. 

2. Shore Duty Personnel 

For those aviation maintenance personnel (E-6 and 

below) assigned to units that do not deploy as a part of a 
Carrier Air Wing Group on a regular basis (e.g., training 
commands such as VF-124 and VAW-110 squadrons which are 
responsible for training maintenance personnel for F-14 and E- 


2 aircraft respectively), MTIP testing is completed annually. 
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Those who have less than six months remaining on a tour need 
not be tested. But again, for the benefit of an individual's 
professional growth, especially if his or her next assignment 
will involve a similar type of aircraft, the squadron 
Maintenance officer should continue to identify weaknesses 
and to improve technical knowledge and skill. 
3. Newly Assigned Personnel 

Newly assigned personnel (E-3 and below) who have not 
completed Fleet Readiness Aviation Maintenance Personnel 
(FRAMP) pipe-line training are tested within six months of 
reporting to a maintenance production work center. This time 
requirement is not applicable to personnel assigned Temporary 
Duty (TAD) as a barracks compartment cleaner or as a member of 
the First Lieutenant Division responsible for cleanliness of 
squadron spaces. Temporary Assigned Duty may last an average 
of three months or more depending on the shortages of junior 
personnel. Therefore, it is possible that these personnel 
will not become part of the Maintenance Training Improvement 
Program until nine or more months after checking into a 
squadron. 

All other newly assigned personnel are tested within 
thirty days of assignment to a maintenance department work 
center. If an individual has completed FRAMP completion tests 


(CF) within 30 days of reporting to a work center, the test 
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results may be used to satisfy the initial MTIP testing 


requirements. 


C. TASK AREAS TO BE TESTED 

The composition and number of work centers vary according 
te the type of aircraft an activity maintains and operates. 
For example, Radar Fire Control Technicians (AQ) and Aviation 
Ordnance Technicians (AO) are not needed in an E-2 squadron. 
Even though work centers such as Aviation Administration, 
Aviation Storekeepers, Quality Assurance, Power Plants, 
Airframes, Parachute Riggers, Environmental Control, 
Electronics, Electric, and Plane Captain/Trouble Shooters are 
common to both F-14 and E-2 squadrons, system knowledge and 
skills required of the respective technicians differ 
considerably from one aircraft to another. Therefore, MTIP 
tests and lesson guides are tailored accordingly to meet an 
activity’s requirements. They are based on applicable 


technical manuals, work unit codes, the Naval Aviation 


Maintenance Program (OPNAVINST 4790.2E), and other 


instructions. All test questions and lesson guides have a 
valid reference or training source sailors can study. 

Areas of knowledge required of a technician can be 
numerous. For example, an Airframe Technician is required to 
be familiar with task areas in the landing gear system, 
hydraulics system, aileron system, rudder system, brake 


system, and pneumatic system. Each task area for each 





occupational specialty is included in the MTIP test. An E-2 
squadron alone consists of approximately 121 task areas to be 
tested and managed. Every task area must meet the minimum 
standard set forth by the functional wing Maintenance Training 
Team. Depending on the skill level, past training, and 
experience level at the time of MTIP testing, it is possible 
for a squadron to accumulate three to four hundred combined 


deficiencies relative to the minimum standards. 


D. REFRESHER TRAINING REQUIREMENTS 
1. Minimum Test Scores 
MTIP remedial training is required for all test areas 
in which scores are less than the previous year’s functional 
wing average or sixty-three percent (whichever is higher). 
Depending on the task area, the minimum cut-off average score 
a be set as high as seventy-five percent by the functional 
wing. Those individuals who do not meet the cut-off grades 
are considered deficient in those task areas. Refresher 
training is also available to those who have successfully 
attained scores above the minimum standard but wish to improve 
or maintain their knowledge of the latest information. 
2. Types of Refresher Training 
a. Scheduled Training 
Formal MTIP refresher training is the primary 
method used to correct documented deficiencies. Subject 


Matter Experts (SME) and other highly experienced technicians 
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from Fleet Readiness Aviation Maintenance Personnel (FRAMP), 
Naval Aviation Engineering Services Unit (NAESU), and Naval 
Air Maintenance Training Group (NAMTD) are designated as 
training instructors. Quarterly MTIP training classes are 
scheduled and published by the functional wing Maintenance 
Training Team. Quotas for training classes are assigned as 
requested by squadrons. Due to the large number of task areas 
to be covered, a particular training class may or may not be 
repeated within the same quarter. But over time, every task 
area will be covered as many times as possible to satisfy 
every squadron’s needs. 

Upon completion of refresher training, the MTT ensures 
retesting is conducted for each individual as soon as the 
class is completed. Additional refresher training is required 
if the minimum standard is not met. 

b. In-house Training 

MTIP lesson guides similar to those used by the 
designated instructors are available in computer disks and 
distributed to each squadron for conducting refresher training 
at their own convenience. Reviews and changes to these lesson 
guides are coordinated by the MTT. Subject matter experts 
from FRAMP, NAESU, NAMTD and the squadrons play a vital role 
in keeping these lesson guides accurate and current. Using 
these lesson guides, training can be conducted anywhere and 


anyplace that operational tempo permits, regardless of whether 
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a squadron is on an aircraft carrier for work-ups or ona 
detachment to a remote training ground. Training quality with 
this method can be comparable or better than the formal 
refresher training at the home base if the subject matter 
experts from NAES'! or the contractors accompany the squadron 
as a member of the detachment. Retesting can be administered 
on site or upon returning to the home base but tests can only 
be graded by the functional wing Maintenance Training Team. 
Testing materials are treated with the utmost security and 
confidentiality. 
c. Integrated Weapons Systems Review (IWSR) 

One other very useful and effective method of 
hands-on training, mostly applicable to the Electrical and 
Avionics ratings, is the Integrated Weapons System Review 
(IWSR) . With squadron aircraft assigned strictly for 
maintenance training, maintenance technicians can proceed 
through the learning process with the help of subject matter 
experts from NAESU or the contractor. Due to the involvement 
of operational aircraft along with outside technical experts, 
IWSR requires tremendous planning and scheduling effort. 
Therefore, only the more experienced technicians are normally 
assigned to this type of training. Assigned aircraft and 
technicians involved in IWSR are not available for the day-to- 
day operational commitments of a squadron which may last as 


long as two to three weeks. IWSR is normally scheduled 
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towards the latter part of a training cycle. IWSR may be 
considered the climax of turnaround (from completion of a 
deployment to the start of a next deployment) training effort, 
signaling the nearness of an upcoming extended deployment. 
The experience gained by these few assigned technicians during 
the IWSR is expected to be passed on to the rest of work 
centers. 
3. MTIP Completion Rates 

Pre-deployment tests for sea duty personnel and annual 
tests for shore duty personnel mark the end of a training 
cycle and freeze the percentage score of an aviation unit. The 
MTIP completion rate is tie number of deficiencies corrected 
divided by the total number of CM and CF deficiencies times 
100. For example, if a squadron completes 293 of the combined 
CM and CF tntal of 470 deficiencies during a turnaround 
training cycle, it is said to have attained a 62 percent MTIP 
completion rate. A minimum of seventy-five percent of 
deficiencies noted in the post-deployment (CM) and FRAMP 
completion (CF) tests is to be corrected using scheduled 
training and/or in-house training (both described earlier in 
this chapter) during the turnaround training cycle. 

4. MTIP On-the-Job Training (OJT) 

On-the-Job (OJT) is the practical instruction of 

personnel in the performance of maintenance tasks. Under the 


supervision of qualified personnel, the trainee learns to 
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complete tasks using an appropriate maintenance manual. 
Experienced personnel are used to instruct, demonstrate and 
impart their skills to the less experienced. Trainees learn 
by observing job performance and gain experience by 
participating in the work effort. OJT syllabi are prepared to 
complement MTIP task areas. Attainment of a task area score 
at or above the cut-off on either the MTIP test or refresher 
training test is a prerequisite to final MTIP OJT 
qualification in a task area. When an individual completes 
MTIP OJT qualification, he(she) is considered fully qualified 
to perform the task indicated. This qualification will remain 
in force until: 

1. A score, on a subsequent CM MTIP test, indicates a 
requirement for refresher training on those task areas with 
scores below the cut-off. 


2. Transfer to an another unit. 


E. MTIP TRAINING REPORTS 

Aviation Training Support System (ATSS) is a Manpower, 
Personnel and Training (MPT) Automatic Data Processing (ADP) 
system that provides principal support for FRAMP, NAMITD, 
Aircraft Intermediate Maintenance Department, and operational 
fleet squadrons during the turnaround training cycle 
(COMFITAEWWINGPACINST 4500.3). 

ATSS is used for MTIP data collection, testing and 


reporting. Access to the MTIP data base is controlled by user 
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identification codes and passwords. Currently, ATSS has 
seventeen separate reports (see Appendix A) available under 
the Maintenance Training Improvement Program. Most of these 
reports, especially those involving test question data, are 
not available to an aviation unit. The most commonly used 
reports directly accessible by a squadron are: 

M01 MTIP Personnel Training Report 

M02 MTIP Work Center/System Training 

MO5 MTIP Squadron/Work Center CDP Report 

M10 MTIP Squadron Training Requirements Reports 

M21 Squadron Turnaround Training Progress Report 
With these reports, a squadron training officer or chief petty 
officer has all the information needed to concentrate training 
effort where it is most needed. He or she has the option to 
go beyond the MTIP refresher training by sending those 
individuals back to formal schools offered by other training 


commands such as NAMTD. 


F. SUMMARY 7 

MTIP has become an integral part of the overall aviation 
Maintenance training program in the Navy. It is a training 
Management system used to identify training deficiencies at 
the various levels, ranging from an individual to the aviation 
community as a whole. Through diagnostic testing procedures, 


a qualitative assessment of individual technical knowledge is 


made for all aviation maintenance personnel. Based on this 
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information, the training program is tailored to an 
individual’s needs and concentrates efforts where they are 
most needed. The completion rates for each aviation squadron 
are used to measure the overall effectiveness of Maintenance 
Training Improvement Program. However, the effects of a 
training program should reflect on unit performance and 
operational readiness, which provide the final measure of 
effectiveness and represent the end products of the 
maintenance training system. 

The next chapter examines the effects of MTIP on F-14 and 
E-2 squadrons at Naval Air Station, Miramar, California. 
Using the MTIP completion rates and information collected 
under the Naval Aviation Logistics Data Analysis (NALDA), 
analyses are conducted on the relationships between training 


and operational readiness. 


a3 


IV. METHOD AND RESULTS 


A. INTRODUCTION 

This chapter discusses the method, participants, 
Materials, procedures, and limitations of this study. It also 
presents the data that are used in analyzing relationships 
between Maintenance Training Improvement Program (MTIP) 
completion rate and operational readiness measures such as 
Full Mission Capable (FMC) rate, number of no-defect (A-799) 
Maintenance actions, and Direct Maintenance Manhours per 


Flight Hour (DMMH/FH). 


B. METHOD 

Training is the systematic acquisition of skills, rules, 
concepts, or attitudes that result in improved performance in 
another environment (Goldstein, 1993). In order to capture 
these two environments, a measure comparing performance before 
training and an assessment of on-the-job performance following 
training are needed to evaluate the effects of the Maintenance 
Training Improvement Program. 

As discussed in Chapter II, a variety of experimental 
designs are available and of course, the ideal model is one 
that would produce "clean" results, having eliminated ali 
threats to validity. But most real-world training occurs in 


settings where perfect evaluation with an ideal model is 
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rarely possible. Due to the inherent nature of military 
service, with the necessity to undertake operations on short- 
term notice, an aviation squadron may be called into action 
thousands of miles away from home base for an extended period 
of time. It is not practical to collect data under test 
conditions that need to be designed and set up each time 
information is needed. It is logical to seek a methodology 
based on practicality and on existing available information to 
track long-term trends in unit performance and combat 
readiness. 

Therefore, the experimental design needs to be tailored to 
accommodate the constraints of the training and/or on-the-job 
setting. A desirable design would differentially control as 
Many threats to internal and external validity as possible, 
and minimize constraints imposed by the environment and the 
influences of the multitude of organizational variables. 
While not perfect, the data collected under these conditions 
makes it possible to avoid a useless evaluation. As Peterson 


(1978) suggests, 


It is better to approximate evaluation than neglect it 
altogether due to its difficulties. For example, 
measuring learning not only at the end of the course, but 
sometime later is better than nothing just because you 
can’t get measures of behaviors or results. 


Based on the concepts discussed in Chapter II, and to 


accommodate a real-world, on-the-job setting as explained 
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above, the locally available MTIP records combined with the 
Navy’s existing maintenance data base from Naval Aviation 
Logistics Data Analysis (NALDA) were selected for this study. 
These data provide a practical approach to measurement, and 
also reflect the ultimate outcome that one would expect from 
a training program. 

MTIP is conducive to the one-group pretest/posttest 
evaluation method. All maintenance personnel (like an 
experimental group) assigned to each squadron are given a 
pretest (post-deployment test) to establish the baseline of 
training deficiencies, then presented with the training 
program, and finally given a posttest (pre-deployment test). 
The MTIP completion rate (Chapter III, section D, subsection 
3) represents the final effort of the training program in 
terms of percentage completed during the allotted time period. 
Appendix B lists all squadrons’ completion rates for this 
study. Newly assigned personnel and those scheduled to be 
transferred before the next deployment are automatically 


excluded from the MTIP completion rate calculation by the 


Aviation Training Support System (ATSS) computer data 


management program. This method provides a measure of 
comparison between before and after training for the same 
group of personnel. Unfortunately, without permission from 
the highest level of authority in the Navy, and also for 
obvious practical reasons, control groups are not feasible for 


the purpose of this evaluation process. Therefore, it is 
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difficult, without a control group, to ascertain that the 
training program is the prime factor determining any 
differences that occur between measurements. 

On the other hand, the Naval Aviation Logistics Data 
Analysis (NALDA) System provides a data base for fleet 
maintenance and operations. It provides information on unit 
performance and operational readiness of an aviation unit. 
Full Mission Capable (FMC) rate, number of no-defect (A-799) 
maintenance actions, and Direct Maintenance Manhours per 
Flight Hour (DMMH/FH) were extracted from the NALDA system; 
they are listed in Appendices C and D. The FMC rate is a 
percentage figure based on the monthly total available 
equipment in-service hours that all systems are fully 
functional to perform all missions designed for an aircraft. 
A-799 actions are the number of maintenance actions involving 
components removed by the Organizational Maintenance Level 
("O" Level) technicians as defective that are found to operate 
properly when tested later. DMMH/FH is the average amount of 
direct maintenance manhours required to support each flight 
hour. These operational measures reflect what should be 
learned as a result of a training program and answers the 
question of improved performance in another environment, on- 
the-job-setting. As noted by other researchers, 

The effects of training on unit performance and 
operational readiness provide the final measure of 
training effectiveness and represent the end products of 


the maintenance training system (Gibson and Orlansky, 
1986, pp.48). 
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One advantage to using information from the NALDA system 
is that it provides an objective measure of real maintenance 
performance. Since data bank information is collected 
routinely and unobtrusively, it represents actual performance 
as opposed to data collected under test conditions that may be 
subject to the "Hawthorne Effect" (Gibson and Orlansky, 1986, 
pp.26). 

However, the disadvantage is that the service's 
maintenance data was not specifically designed for training 
purposes. To be useful for evaluating the effect of training 
on an individual, it is necessary to be able to clearly relate 
the maintenance data to the specific system, work center, and 
the performing technician. The NALDA information does not 
identify individual(s) who was(were) actually involved in the 
performance of a maintenance task. Therefore, the objective 
measures of performance collected by this method only reflect 
the group as a whole. 


Correlation analyses are conducted between the results of 


the MTIP completion rates and the selected performance 


measures from the NALDA data. It is reasonable to believe 
that well-trained personnel perform better on their jobs than 
less-well-trained personnel. It is assumed here that higher 
MTIP completion rates are related to better-trained personnel. 
If this is so, the effects of better training should be 


observable in the selected NALDA operational measures. 





1. Participants 

During the planning phase of this study, the author 
examined the types of aircraft that are deployable aboard an 
aircraft carrier. The west coast carrier deployable aircraft 
are home-based at several Naval Air Stations (NASs) in the 
State of California, including NAS Lemoore, NAS North Island 
and NAS Miramar, and at NAS Whidbey Island located in the 
State of Washington. Preliminary information was requested 
from the Maintenance Training Improvement Program Offices and 
the Data Analysts of the Pacific Fleet Light Attack Wing 
(LATWINGPAC) at NAS Lemoore, Anti-Submarine Warfare Wing 
(ASWWINGPAC) at NAS North Island, and Fighter Airborne Early 
Warning Wing (FITAEWWINGPAC) at NAS Miramar. The author 
understands the extra workload that such requests place on 
personnel at these units. Thus, it is not surprising that 
cooperation was sporadic. However, the MTIP Office and the 
Data Analyst from the FITAEWWINGPAC have been extremely 
cooperative and helpful. Therefore, F-14 (Tomcat) and E-2 
(Hawkeye) squadrons under the Fighter Airborne Early Wing, 
Pacific Fleet at Naval Station Miramar were selected as the 
subjects of this thesis. 

Once the types of aircraft were selected, the emphasis was 
placed on the sea-going squadrons to be consistent with the 
types of operational cycles involved (e.g., post-deployment, 
turnaround training, and deployment). Ten F-14 squadrons and 


six E-2 squadrons are considered to be sea-going units. 
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However, two F-14 squadrons and one E-2 squadron are currently 
assigned to USS Independence, homeported at Naval Station 
Yokosuka, Japan. These three squadrons are excluded from this 
study because of their unique operational tempo, which is 
quite different from the rest of the squadrons. Therefore, 
the total number of squadrons involved in this study is 
thirteen. 

An average of 180 and 110 enlisted maintenance technicians 
are respectively assigned to each F-14 and E-2 squadron. 
After subtracting fifteen percent (a conservatively high 
estimate) of E-7 and above enlisted personnel from each 
squadron, the number of E-6 and below maintenance technicians 
is estimated to be around seventeen hundred. 

Since all maintenance personnel (E-6 and below) are 
required to be tested under the Maintenance Training 
Improvement Program, the age bracket of these personnel ranges 
from 19 years old to the early 30s. There are no females 
currently assigned to sea-going squadrons deployable aboard 
aircraft carriers in the Navy. With recent changes in the 
political climate, this will certainly change in the near 
future. 

Due to the method of data collection from the two 
centralized sources, none of these maintenance personnel, 
squadrons, nor the FITAEWWINGPAC staffs are aware of this 


experiment. This is done to hopefully control some of the 
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internal and external variables which may influence the 
evaluation process. 

2. Materials Used 

a. Source of Data 

The data for this study came from two of the Navy's 
data banks: Aviation Training Support System (ATSS) and Naval 
Aviation Logistics Data Analysis (NALDA) System. Maintenance 
Training Improvement Program records are managed by the ATSS 
located at Naval Air Station Miramar. Operational readiness 
data are obtained from the NALDA System, Patuxent River, 
Maryland via a computer terminal with the assistance of the 
FITAEWWINGPAC Data Analyst at NAS Miramar. 

Due to data processing and storage limitations of ATSS, 
most squadron MTIP records are only kept to cover a period of 
a couple of years. Therefore, the historical files including 
the messages issued monthly by the FITAEWWINGPAC MTIP Office 
during the period of January 1989 to December 1992 are used as 
the source of the MTIP completion rates. As a result of this 
limitation, operational data from NALDA has also been selected 
to match the available MTIP records covering the same period. 

The total number of pages of NALDA information obtained 
for this study is estimated to be about 1200 pages of full 
size computer printouts. Even with the assistance of a 


senior, experienced data analyst, the NALDA data took more 


than two months to collect. First of all, this is partly due 





to the tremendous amount of time required to get connected 
with the NALDA system. The data has to be requested by each 
squadron, and by each month for which it is needed, in the 
format desired and it has to be printed via telephone line 
with computer hook-up to the source. Secondly, the data 
analyst had to do all of this on a voluntary basis and very 
often on his own time. 

b. Hardware and Software 

A standard IBM PS/2 286-based micro-computer, model 60 
with one megabyte of RAM and Disk Operating System (DOS) 
Version 5.0 served as the work station for this study. This 


system is probably considered antiquated compared to what is 


available on the market today. However, for conducting a 


routine statistical analysis, this system serves the purpose 
more than adequately. 

The Lotus 1-2-3 (DOS Version 2.2) spreadsheet software 
program was used to discriminate, sort, and compile the 1200 
pages of raw data to a form relevant for the periods under 
consideration. These files were used as inputs to a second 
software program, Statgraphics (DOS Version 6.0), for the 
correlation analysis. Lastly, WordPerfect (DOS Version 5.0) 


was used to document the entire process. 











C. PROCEDURE 
Now that the methodology, participants, and 
software/hardware have been discussed in detail, the steps in 
the execution of this analysis can be summarized as follows: 
1. Collect MTIP completion rates and NALDA operational 
measures for each squadron. 

2. To ensure comparability, the MTIP data has to be 
compiled in parallel with each squadron’s turnaround training 
cycle (from post-deployment test to pre-deployment test) and 
the NALDA data has to cover the time period following the pre- 
deployment test to the next post-deployment test. The 
assumption is that during deployment (including a month or two 
before and after training) each squadron is provided with 
maximum support from up and down the chain of command. If 
absolutely necessary, the support will come in the form of 
temporarily sacrificing or limiting tthe operational 
capabilities of other similar commands not scheduled for 
deployment. Therefore, squadrons on deployment will most 
likely be as closely comparable as possible. In this manner 
the effects of training are measured with operational 
readiness under similar conditions. 

3. MTIP completion rates represent training efficiency of 
each squadron as a whole with one number representing the 
total effort of the entire turnaround training. However, 
operational readiness figures, which are also group measures 


for each squadron, are collected on a monthly basis throughout 
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the deployment. Therefore, the operational readiness data 
needs to be averaged out so that it may be compared with the 
MTIP data. To ease the management of these data, Lotus 1-2-3 
was used to arrange/rearrange, compile, and calculate the 
information required. 

4. Import the compiled data from Lotus 1-2-3 to the 


Statgraphics program and conduct correlation analyses. 


D. LIMITATIONS 

1. Available Data 

As mentioned earlier in this chapter, non-availability 

of the MTIP data over an extended period of time limits this 
study to a period of four years. Considering that most of the 
MTIP offices and the Aviation Training Support System (ATSS) 
retain information for only a couple of years, it is fortunate 
to be able to obtain the data covering the period from Jan 
1989 to Dec 1992. This amount of data would seem sufficient 
to conduct an assessment of the Maintenance Training 
Improvement Program (MTIP). 7 

However, squadrons deploy at different times during that 
four years. Some squadrons have a much longer turnaround 


training cycle than others, due to the transition to a newer 


model/type of aircraft or major configuration changes to the 


existing ones. A few squadrons are permanently stationed in 
Japan with different operating cycles than those in the 


continental United States. Some were in the middle of a 








deployment when the December 1992 data were collected. For 

these various veasons, on an average, each squadron has only 

one comolete set of MTIP and NALDA data usable for this study. 
2. Internal and External Validity 

Due tc the four-year period of time involved, specific 
events other than training occurring between sequential 
measurements were not subject to control. Biological or 
psychological effects that systematically vary with the 
passage of time as participants become older, fatigued, or 
more or less interested in the job between measurements were 
also not subject to control. 

Influence of the pretest on the scores of the posttest 
and the reactive effects of the pre-deployment test which can 
lead to increased sensitivity to the instructional procedure 
were negligible because of the many months in between each 
test (6 to 8 months for post-deployment test and 12 to 18 
months for pre-deployment test). 

Rivalry between squadrons does exist. Sister 
squadrons that have the same deployment schedule under the 
same Air Wing often put up extra efforts to outperform the 
other. The MTIP office makes unannounced visits to those 
squadrons that are either doing too well or not too well to 
substantiate the reasons behind the success or failure. Self- 


serving, innovative ways of improving the MTIP completion rate 
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is thus kept to a minimum. It is impossible to consistently 
manipulate completion rates or operational readiness measures. 

Influences of the interactions between variables upon 
an evaluation are more difficult to understand and control. 
It is not clear as to how, and in what ways, such interactions 
affected this evaluation. 

Other variables (see Chapter II for an explanation of 
these threats) such as instrumentation, differential selection 
of participants, and reactive effects of experimental settings 
are well under control. The Maintenance Training Improvement 
Program (MTIP) is mandatory for all E-6s and below. Each 
squadron operates under the same policy and guidance set forth 
by the highest level of the chain of command with the MTIP 
office serving as the monitoring activity. Squadrons going 
through major configuration or model changes are retested by 
the MTIP office based on the new information and are 


selectively excluded from this study. 


E. RESULTS 7 

Based on the consolidated data collected from the eight 
F-14 and five E-2 squadrons, this section provides the 
correlation analyses between the MTIP completion rates and the 
selected operational measures. The procedure generates a 
Matrix of correlation coefficients for the observed values, 


and provides a preliminary view of the relationships among 


variables. The strength of the reJationship between variables 
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are evaluated using the Pearson Product Moment Correlation 
Coefficient. The coefficient values fall between -1 and +1. 
A positive correlation indicates that the variables vary in 
the same direction while a negative correlation indicates that 
the variables vary in the opposite direction. Statistically 
independent variables have an expected correlation of zero. 
Tables 1 and 2 provide a synopses of the F-14 and E-2 
aircraft community consolidated data, respectively, collected 
by each squadron on the MTIP completion rate, Full Mission 
Capable (FMC) rate, number of no-defect maintenance (A-799) 
actions, and Direct Maintenance Manhours per Flight Hour 
(DMMH/FH). In order not to disclose the name of each aviation 


unit, numbers are used to represent squadrons. 
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MTIP 
Completion Capable 
Rate Rate 
(Percent) (Percent) 


Note: Detailed F-14 squadrons month-by-month operational 
figures can be found in Appendix C. 





= J 


Source: Compiled from the MTIP Office historical files and NALDA data. 
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Full Mission 











Completion Capable 
Rate Rate 
(Percent) (Percent) 





Note: Detailed E-2 squadrons month-by-month operational 
figures can be found in Appendix D. 










Source: 


Table 3 provides a general overview CE the 
intercorrelation statistics obtained, based on the procedure 


discussed in this chapter. 


TABLE 3 INTERCORRELATION MATRIX 






Maintenance Training Improvement Program 
FMC Full Mission Capable 

A-799 Number of no-defect maintenance actions 
Direct Maintenance Manhours/Flight Hour 





1. Relationship of MTIP to FMC 

Correlation between the Maintenance Training 
Improvement Program (MTIP) completion rate and the Full 
Mission Capable (FMC) rate shows that there is a moderate 
positive relationship between the two variables. As reflected 
in Table 3, the correlation coefficient is 0.61. The 
probability of a relationship this strong occurring by chance 
is 0.03 as determined by a t-test. In other words, the Full 
Mission Capable rate is positively associated with the MTIP 
completion rate. As the MTIP completion rate increases, the 
FMC rate also tends to increase in the same direction. 
However, this does not mean that there is a causal 
relationship between the two variables. We can only infer 
that the two variables have a moderate tendency to increase or 
decrease simultaneously. 

2. Relationship of MTIP to A-799 

Correlation between the Maintenance Training 
Improvement Program (MTIP) and the number of no-defect (A-799) 
maintenance actions produces a weak negative relationship. 
The MTIP completion rate and the number of A-799 maintenance 
actions have a correlation coefficient of -0.26 (see Table 3). 
The correlation is statistically insignificant. The 
probability of a relationship occurring by chance between 
these two variables is about 0.39. A negative correlation 


coefficient suggests that the variables tend to increase or 
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decrease in the opposite directions simultaneously. However, 
a negative correlation between the MTIP completion rate and 
the number of no-defect maintenance actions is expected. Non- 
faulty parts are those that were removed but found not to be 
defective when tested later. Previous research does offer 
some evidence that maintenance technicians may conduct 
Maintenance in a more appropriate and efficient manner as 
maintenance technicians learn through a training program and 
become more knowledgeable. Thus, the results found here are 
in the expected direction but not strong enough to be 
conclusive. 
3. Relationship of MTIP to DMMH/FH 

Even though it is reasonable to believe that with more 
training the time required to complete a maintenance action 
will be reduced, in this comparison, there does not appear to 
be any relationship between the MTIP completion rate and the 
Direct Maintenance Manhours per Flight Hour (DMMH/FH). The 
MTIP completion rate and the DMMH/FH have a negative 
correlation coefficient of -0.02. The probability of a 
relationship occurring by chance between these two variables 
is about 0.94. In other words, MTIP and Direct Maintenance 
Manhours per Flight Hour are statistically independent witha 
correl .ion coefficient of nearly zero. 

As reflected in Table 3, the correlation between the 


Direct Maintenance Manhours per Flight Hour (DMMH/FH) and the 
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number of A-799 maintenance actions provides positive but not 
statistically significant evidence that as the number of 
inappropriate or inefficient maintenance practice increases, 
the Direct Maintenance Manhours per Flight Hour also increases 
in the same direction. The correlation coefficient is 0.36 
and the probability of a relationship this strong occurring by 
chance between these two variables is about 0.23. 

The correlation between the DMMH/FH and the Full 


Mission Capable (FMC) rate produces a moderately negative 


relationship. As expected, when the DMMH/FH decreases, the 


FMC rate improves. The correlation coefficient between the 
DMMH/FH and the FMC rate is -0.45. And, the probability of a 
relationship this strong occurring by chance between these two 
variables is at about 0.12. 

Therefore, as seen in this section, the Direct 
Maintenance Manhours per Flight Hour could possibly be 
indirectly related to the MTIP completion rate through a third 
variable such as the number of A-799 no-defect maintenance 
actions or the FMC rate. The relationships between the MTIP 
completion rate, the FMC rate, the number of no-defect 
maintenance actions and the DMMH/FH are summarized graphically 


in Figure 1. 








r=-0.26 


Figure 1. 





r=0.36 


Intercorrelation Diagram 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 


Training is defined as the systematic acquisition of 
skills, rules, concepts, or attitudes that result in 


improved performance in another environment (Goldstein, 
1993). 


The purpose of the Maintenance Training Improvement 
Program is, therefore, to prepare sailors from the training 
environment to perform the technical tasks necessary in the 
real-world environment to assure the availability and proper 
functioning of aircraft weapons systems. The ultimate goal is 
to improve unit performance in terms of operational readiness. 
Without systematic evaluation, there is no feedback to provide 
information necessary to enhance and improve this training 


program. 


Within the context of the limitations discussed earlier, 


the results of this thesis have provided empirical support for 
the idea that aviation maintenance training under the MTIP 
management system may have an impact on unit performance. The 
MTIP completion rate does show a tendency to increase or 
decrease simultaneously with one of the most important 
operational readiness measures--the FMC rate. The MTIP 


completion rate is a moderate predictor of the Full Mission 








Capable rate; as more training is accomplished, the FMC rate 
tends to improve in the positive direction. 

The MTIP has also been shown to have a negative 
relationship with the total number of no-defect maintenance 
actions performed by the maintenance technicians. Well - 
trained maintenance technicians are less likely to conduct 
maintenance in an inappropriate and inefficient manner. 
Therefore, fewer non-faulty parts will be removed and found to 
be operational later. 

The MTIP completion rate appears to have no correlation 
with the Direct Maintenance Manhours per Flight Hour 
(DMMH/FH). However, as reflected in Table 3 and Figure 1 of 
Chapter IV, the MTIP completion rate is moderately and 
positively correlated to the FMC rate, and only weakly and 
negatively correlated to the A-799 actions. But at the same 
time, the DMMH/FH reveals that it is also moderately 
correlated to the A-799 actions and the FMC rates. Therefore, 
it is possible that the MTIP completion rate is somehow 
indirectly related to the DMMH/FH through a third variable 
such as the number of A-799 maintenance actions or the FMC 
rate. This is an area that could be explored further. 

In interpreting the results of the research conducted for 
this thesis, using correlation analyses, some insight has been 
gained in explaining the question raised in the introductory 
chapter on whether the effects of training are observable in 


operational indicators. 











B. RECOMMENDATIONS 

This thesis has just touched the surface of evaluating the 
Maintenance Training Improvement Program and has disclosed 
several areas where there is room for more in-depth follow-up 
research. Hopefully, this paper will serve as the catalyst to 
ultimately enhance and improve the aviation training program 
under the MTIP management system. 

Until now, the effectiveness of MTIP training has been 
evaluated primarily by end-of-course tests with the MTIP 
completion rate as the final group measure. Unless we have 
credible information about how well technicians perform after 
training in the real-world-setting, we do not know very much 
about whether the course provided the information needed to 
perform well on the job. This thesis has introduced some of 
the available operational measures to link training with 
operations. However, to better evaluate MTIP, operational 
measures should be directly traceable to an individual instead 
of a group. Modifications to the existing NALDA data base may 
be required. Use of local historical files is another option 
to trace an individual’s training and his/her on-the-job 
performance. Measures other than the MTIP completion rate, 
such as job-sample tests and on-equipment performance tests, 
also need to be explored. 

As observed in this study, collection and processing of 
the required data can be extremely time consuming. With some 


training and proper software packages, the Data Analysts and 


56 











the Wing Training Officers could systematically collect the 
necessary data on a continual basis. They could also perform 
simple analyses on their own to draw inferences, revealing the 
possible weaknesses and strengths of a training program, or 
make these data available for further study by others like 
students at the Naval Postgraduate School. 

It is recommended that the Aviation Training Support 
System (ATSS) database storage capacity and the local MTIP 
Office files be expanded to retain historical files over a 
longer period of time. It will be extremely useful to be able 
to track training performance over several complete cycles of 
training and deployment. 

It is not clear at this point why the MTIP completion rate 
and the Direct Maintenance Manhours per Flight Hour did not 
produce the expected correlation. On one hand, the result may 
be a factor of the particular data captured here and not 
generalizable to other periods of time or other squadrons. On 
the other hand, if the data reflect weakness in the training, 
this is an important issue. Another possibility is that the 
zero correlation is a consequence of mixing data for two 
different aircraft types. Further research should be 
conducted to explore this finding. 

Another research area that should be explored is the 


correspondence between the objectives of MTIP and actual job 


performance. Is there 100 percent correspondence between what 





is taught and what is expected on the job? Until this 
analysis is conducted, further evaluation may be meaningless. 

In view of the results of this thesis in empirically 
linking the west coast F-14 and E-2 aircraft community 
Maintenance Training Improvement Program (MTIP) completion 
rates with the operational readiness measures s’.ch as the Full 
Mission Capable (FMC) rates and the number of no-defect (A- 
799) maintenance actions, the next logical step would be to 
conduct a study involving all types of aircraft in the Navy. 
Other exploratory studies will add information that can be 
used to ultimately determine the training effectiveness of 
MTIP. Resources allocated to MTIP and personnel safety make 


adequate evaluation critical. 
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APPENDIX A 


AVAILABLE MTIP REPORTS 






ee es 
| Moz | rtp Work Center/ System Training | 
[mos | rtp class status 
[moa | rtp Course summary Report 
[wos 






MTIP Squadron/Work Center/CDP Report 


MTIF Wing Test Area Report by Work 
Center/System 

MTIP Squadron Test Area Report by Work 
Center/System 

MTIP Wing Work Center/System Report by 
Squadron 


MTIP Wing Test Area Report by Squadron 


MTIP Squadron Training Requirements 
Reports 


MTIP Top Wing System Training | 
Requirements 
ora MTIP Gradebook Random Report 







MTIP Test Item Data Random Review 
MTIP Work Center/System Report 


MTIP Post/Pre-Cruise Training Progress 
Report 

Functional Wing Refresher Training 
Progress 

Squadron Turnaround Training Progress 
Report 
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APPENDIX B 


MTIP COMPLETION RATES 


F-14 rons MTIP TE 
1 57 
2 75 
3 73 
4 75 
5 31 
6 65 
7 50 
8 62 

E-2 S drons MTIP_ RATES 
1 37 
2 66 
3 41 
4 72 
5 90 
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OcT 





APPENDIX C 


F-14 OPERATIONAL DATA 


Squadron No.1 


Squadron No.2 


A-799 


A-799 


DMMH / FH 


DMMH / FH 





Squadron No.3 


DATE %FMC A-799 DMMH / FH 
1990 
FEB 60 23 73 
MAR NA 42 NA 
APR 47 64 31 
MAY 30 59 51 
JUN 52 75 57 
JUL 73 64 48 
AUG 77 98 63 
SEP 62 102 51 
OCT 59 96 59 
NOV 69 49 86 
DEC 84 30 59 
Avg 61 64 58 


Squadron No.4 


DATE % FMC A-799 DMMH / FH 


1990&91 





*Squadron No.5* 


A-799 DMMH / FH 


Note: Squadron No.5 is assigned to USS Independence. 


Squadron No.6 


DATE % FMC A-799 DMMH / FH 


ss 5525252555525 2525 2552525255525 25525222 5252525 =5=>= 


1989&1990 





Squadron No.7 


DATE $ FMC A-799 DMMH/ FH 
1991 
MAR 15 70 48 
APR 29 51 47 
MAY 44 49 27 
JUN 50 46 68 
JUL 29 85 63 
AUG 27 72 43 
SEP 32 59 40 
OCT 46 43 51 
NOV 61 18 62 
DEC 23 8 116 
Avg 36 50 56 


*Squadron No. 8* 


DATE %FMC A-799 MMH / FH 
1990 
FEB 50 57 40 
MAR 100 56 44 
APR 33 54 44 
MAY 61 109 36 
JUN 31 56 49 — 
JUL 29 44 47 
AUG 34 21 46 
SEP 50 40 44 
OocT 68 50 35 
NOV 75 63 55 
DEC 88 37 40 
Avg 56 587 43 


Note: Squadron No.8 is assigned to USS Independence. 
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Squadron No.9 


DATE + FMC A-799 DMMH / FH 
1990&91 

DEC 23 28 31 
JAN 26 68 29 
FEB 26 35 49 
MAR 54 61 24 
APR 54 66 22 
MAY 47 61 23 
JUN 58 89 27 
JUL 35 57 29 
AUG 72 25 33 
SEP 33 30 26 
Avg 43 52 29 


Squadron No.10 


DATE %FMC A-799 DMMH / FH 
1991 

MAR 44 64 47 
APR 25 63 46 
MAY 36 42 35 
JUN 44 43 58 
JUL 43 73 70 
AUG 18 50 60 — 
SEP 24 39 49 
OCT 37 29 85 
NOV 55 19 91 
DEC 25 8 377 
Avg 35 43 92 
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APPENDIX D 


E-2 OPERATIONAL DATA 


Squadron No.1 


DATE %FMC A-799 DMMH / FH 
1991 

JAN 40 51 19 
FEB 29 26 29 
MAR 37 69 18 
APR 20 55 24 
MAY 31 68 30 
JUN NA 83 NA 
JUL NA 33 NA 
AUG 35 19 21 
SEP 18 6 38 
Avg 30 46 26 


Squadron No.2 


DATE FMC A-799 DMMH/ FH 
1990 
FEB 42 22 38 
MAR 100 NA 26 
APR 46 44 25 ~ 
MAY 53 24 18 
JUN 53 21 26 
JUL NA 48 NA 
AUG 44 44 24 
SEP NA 62 NA 
OCT NA 34 NA 
NOV 66 24 19 
DEC 73 17 16 
Avg 60 34 24 
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Squadron No.3 


DATE % FMC A-799 DMMH / FH 
1989&90 
NOV 82 19 24 
DEC 62 29 29 
JAN 75 36 25 
FEB 72 49 24 
MAR 71 67 15 
APR 48 39 15 
MAY 53 55 20 
JUN 75 42 25 
Avg 67 42 22 


*Squadron No.4* 


DATE % FMC A-799 DMMH / FH 
1990891 
DEC 75 42 35 
JAN 22 19 71 
FEB 32 57 33 
MAR 44 32 18 
APR 38 31 23° 
MAY 57 39 59 
JUN 55 43 21 
JUL 75 38 24 
AUG 77 37 19 
SEP 73 37 25 
ocT 99 7 108 
NOV 43 24 63 
DEC 72 10 23 
Avg 59 416 40 


Note: Squadron No.4 is assigned to USS Independence. 
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Squadron No.5 


DATE FMC A-799 DMMH / FH 
1990&91 

OCT 51 21 19 
NOV NA 26 NA 
DEC 47 25 24 
JAN 50 4 14 
FEB 46 2 13 
MAR 52 26 19 
APR 55 15 19 
MAY 73 23 21 
JUN 85 1 56 
Avg 57 16 23 


Squadron No.6 


DATE *%FMC A-799 DMMH/ FH 
1991 

APR 57 16 13 
MAY 65 23 24 
JUN 75 20 25 ~ 
JUL 66 34 29 
AUG 68 23 22 
SEP 67 18 22 
OcT NA 24 NA 
NOV 80 12 35 
Avg 68 21 24 
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